# English Pre-training

T5-Efficient-SMALL-KV32 is a variant of Google's original T5, adopting a deep narrow architecture focused on improving downstream task performance.

BERT is a transformer model pre-trained on large-scale English corpora through self-supervised learning, utilizing masked language modeling and next sentence prediction objectives

T5-Efficient-SMALL-DM768 is a variant of Google's original T5, adopting a deep narrow architecture that prioritizes increasing model depth to enhance downstream performance.

T5-Efficient-BASE-FF9000 is a variant of Google's original T5, adopting a deep narrow architecture that delivers superior performance on downstream tasks with similar parameter scales.

T5-Efficient-SMALL-KV256 is a variant of Google's T5, optimized for downstream task performance using a deep narrow architecture, with 117 million parameters, requiring fine-tuning for use.

T5-Efficient-MINI is a variant of Google's original T5, adopting a deep narrow architecture that demonstrates superior downstream task performance among models with similar parameter counts.

T5-Efficient-TINY-NL2 is a variant of Google's original T5, adopting a deep narrow architecture focused on enhancing downstream task performance.

T5-Efficient-TINY-NL8 is an efficient variant of the Google T5 model, optimized for downstream task performance using a deep narrow architecture.

A Transformer model pre-trained on English corpus using self-supervised learning similar to ELECTRA, trained by predicting replaced tokens.

T5-Efficient-SMALL-EL2 is a variant of Google's original T5, adopting a deep narrow architecture that prioritizes increasing model depth to enhance downstream performance.

T5 Efficient Large-NH32 is a deep-narrow variant of Google's T5 model, focusing on improving downstream task performance by increasing model depth.

T5-Efficient-BASE-NL48 is a variant of Google T5, adopting a deep narrow architecture that prioritizes increasing model depth to enhance downstream task performance.

XLNet is an unsupervised language representation learning method based on a generalized permutation language modeling objective, using Transformer-XL as the backbone model, excelling in long-context tasks.